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Introduction 


This  is  the  final  progress  report  for  a  joint  project  between  Geisler’s  laboratory  at 
University  of  Texas  at  Austin  and  OWL  Displays  Inc.,  to  develop  a  real  time 
variable  resolution  (foveated)  imaging  system  for  video  communications  tasks 
such  as  remote  control  of  unmanned  vehicles. 

The  original  objectives  in  Phase  I  were  to  develop  fast  and  efficient  software  for 
encoding  foveated  images  (to  be  done  at  UT),  and  to  design  and  construct  variable 
resolution  display  hardware  for  displaying  the  encoded  foveated  images  (to  be 
done  at  OWL).  During  Phase  I  a  software  encoder/decoder  and  a  small  test 
display  were  developed.  However,  the  software  was  so  efficient  that  conventional 
displays  proved  to  be  adequate  for  moderate  sized  images.  Thus,  in  Phase  II  it 
was  decided  that  OWL  would  focus  only  on  developing  a  very  high  resolution 
display,  while  Geisler’s  lab  would  focus  on  improving  the  software 
encoder/decoder  and  on  integrating  it  with  a  heads-free  eye  tracker.  However, 
during  Phase  II  a  complete  redesign  of  the  software  led  to  further  increases  in 
encoding/decoding  speed  and  elimination  of  display  artifacts  that  had  existed  in 
the  earlier  version  on  conventional  displays.  Thus,  it  became  apparent  by  the 
second  year  of  Phase  II  that  the  display  technology  that  OWL  could  provide  was 
not  going  to  significantly  enhance  system  performance.  At  that  point  OWL  (with 
AFOSR  approval)  turned  their  attention  to  eye  tracking  technology  to  be  used  in 
conjunction  with  the  foveated  imaging  software.  OWL  in  cooperation  with  ASL 
Inc.  succeeded  in  integrating  a  small  eye  tracker  into  a  helmet  mounted  display 
(the  Virtual  Research  Systems  Inc.  V8  helmet).  This  display  system  has  been 
integrated  and  tested  with  the  foveated  imaging  software.  During  the  final  year  of 
Phase  II  financial  difficulties  overtook  OWL;  they  filed  for  bankruptcy  and  are 
now  out  of  business. 

Although  the  OWL  side  of  the  project  might  best  be  described  as  a  modest  success, 
we  believe  the  software  and  software/hardware  integration  performed  at  UT  has 
been  a  big  success. 


Original  Objectives 

The  specific  original  Phase  I  objectives  were  as  follows: 

1.  Develop  algorithms  for  foveated  vision. 

2.  Implement  and  test  algorithms  on  hardware  platforms. 

3.  Interface  camera,  eye  tracker  and  CRT  hardware  platform. 

4.  Develop  a  low  cost  monochrome  screen  for  the  foveated  system  based  on 
owl's  technology 

5.  Develop  lattice  application  specific  integrated  circuit  (ASIC)  interface. 

6.  Interface  camera,  eye  tracker  and  OWL's  display  platform 
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7.  Conduct  a  final  comparison  and  assessment  of  achievable  bandwidth 
reduction  at  given  levels  of  display  quality. 

The  specific  original  Phase  II  objectives  were  as  follows: 

1.  Completion  of  a  one-camera  image  acquisition  system. 

2.  Reduction  or  elimination  of  the  apparent  motion  effects  seen  in  the  Phase  I 
FIS. 

3.  Construction  of  a  higher  resolution  OWL  display  using  a  pulse  driver. 

4.  Implementation  of  a  heads-free  eye  tracking  system  (i.e.  one  that  allows 
free  head  movements)  and  evaluation  of  its  robustness. 

5.  Evaluation  of  human  performance  while  using  the  system  (for  both 
conventional  and  OWL  screen  technologies),  for  a  number  of  different  image  types 
and  tasks,  including  visual  search,  text  reading,  and  navigation  through  the 
environment. 

6.  Design  and  implementation  of  a  data  compression  scheme  to  work  in 
conjunction  with  the  FIS  to  further  reduce  bandwidth  requirements. 

7.  Evaluation  of  the  Foveated  Imaging  System's  transmission  properties. 

8.  Evaluation  of  alternate  hardware/software  implementations. 

Status  of  effort 

Phase  I  Objectives  1,  2,  4  and  7  were  completed.  Objective  3  was  completed  except 
for  interfacing  of  the  camera  which  was  accomplished  during  Phase  II. 

Objectives  5  and  6  were  not  completed  because  of  the  change  in  OWL’s  objectives 
described  in  the  Introduction.  Phase  II  objectives  1,  2,  4,  6,  7  and  8  were  all 
completed.  Objective  3  was  not  completed  because  of  another  change  in  OWL’s 
objectives  described  in  the  Introduction.  Objective  5  was  not  completed  in  a 
rigorous  fashion.  Some  behavioral  testing,  demonstrating  the  advantages  of 
foveated  imaging  for  optimizing  search  performance  under  fixed  bandwidth 
transmission  conditions,  was  completed  in  Phase  I  (Kortum  &  Geisler,  1996). 
However,  for  three  reasons  additional  behavioral  performance  testing  was  not 
completed  in  Phase  II.  First,  new  insights  led  to  a  complete  redesign  of  the 
software  algorithms;  these  changes  required  considerable  additional  effort  to 
implement.  Second,  the  development  of  real-time  conventional  image 
compression  algorithms  appropriate  for  foveated  imaging  grew  into  a  much 
larger  effort  than  initially  anticipated.  Third,  it  was  decided  that  higher  priority 
should  be  placed  upon  demonstrating  the  compatibility  of  foveated  imaging  with 
existing  image  compression  standards  such  as  MPEG. 
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Work  on  the  foveated  imaging  system  and  software  libraries  continues  with  some 
limited  funding  from  UT.  We  plan  to  maintain  and  expand  the  software  library 
for  some  years  to  maximize  the  chance  for  applications  to  develop. 

Accomplishments/New  Findings 

The  major  accomplishments  and  findings  are  described  here  with  an  emphasis 
on  work  carried  out  at  UT  in  Geisler’s  lab. 

(1)  A  general  purpose  real-time  C-I-+  library  has  been  developed  for  coding  and 
decoding  variable  resolution  static  images  and  video,  both  in  8-bit  gray  scale  and 
24-bit  color.  This  linkable  library  (for  Pentium  class  computers  running  the 
Windows95/98/NT  OS)  will  soon  be  made  generally  available  for  development  and 
testing  of  applications  involving  variable  resolution  displays.  We  anticipate 
having  the  library  available  with  documentation  within  a  couple  of  months.  Real¬ 
time  demonstration  executables  using  the  library  are  currently  available  at  our 
web  site:  http ://fi . cvis . ps v. utexas . edu . 

The  details  of  the  algorithms  are  described  in  Geisler  &  Perry  (1998)  and  in 
documentation  available  at  the  web  site.  In  brief,  a  low-pass  pyramid  of  5-6  levels 
is  first  created.  Each  successive  level  of  the  pyramid  is  a  copy  of  the  image  from 
the  previous  level,  but  at  1/2  the  resolution  in  each  direction  (1/4  the  number  of 
pixels).  Based  upon  the  foveation  points  (i.e.,  the  locations  of  highest  resolution) 
subsets  of  pixels  from  each  level  of  the  pyramid  are  selected  to  create  a  series  of 
smaller  images.  Typically,  all  of  these  smaller  images  together  contain  only  a 
fraction  of  the  total  pixels  in  the  original  image.  The  subimages  can  then  be 
processed  in  any  fashion,  just  like  normal  images.  For  example,  they  can  be 
MPEG  coded  and  transmitted  to  a  remote  site  and  then  MPEG  decoded.  The 
processed  small  images  are  then  decoded,  interpolated  and  smoothed  by  our 
software  to  obtained  a  displayable  foveated  image. 

(2)  We  have  tested  our  real  time  software  in  conjunction  with  MPEG  (H.263)  and 
shown  that  it  generally  produces  very  substantial  bandwidth  savings  both  for  I 
frames  and  P  frames.  The  results  are  described  in  documentation  available  at  the 
web  site.  The  results  will  appear  in  an  invited  SID  paper  (Geisler  &  Perry,  1999). 
As  implied  above,  our  foveation  software  was  explicitly  developed  for  pre-  and  post¬ 
processing,  and  hence  it  is  generally  compatible  with  a  wide  range  of  image 
processing  hardware. 

(3)  We  have  also  developed  our  own  real  time  image  compression  software  which 
includes  fast  motion  estimation/compensation,  fast  pyramid  coding,  fast  zero-tree 
coding  and  arithmetic  coding.  These  are  conventional  state-of-the-art  components 
of  the  best  current  image  compression  algorithms.  Our  contribution  was  to 
develop  versions  with  very  good  real  time  performance.  We  have  shown  that  our 
foveation  coding  increases  the  speed  of  subsequent  processing  because  of  the  great 
reduction  in  the  number  of  pixels  that  must  be  processed.  With  our  software  both 
variable  resolution  coding  and  more  conventional  image  compression  can  be 
accomplished  in  real  time  for  moderate  sized  images,  without  any  special  purpose 
hardware.  A  real-time  C++  library  of  these  routines  will  also  be  made  generally 
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available  for  development  and  testing  at  or  near  the  same  time  the  foveated 
imaging  library  is  made  available. 

(4)  A  foveated  imaging  web  site  has  been  created  which  contains  much 
information  about  foveated  imaging,  including  downloadable  real-time 
demonstrations.  This  web  site  has  received  many  hundreds  of  hits  (from  unique 
addresses)  since  its  inception  in  September,  1998.  Feedback  about  the 
demonstrations  has  been  very  positive.  A  number  of  laboratories  have  indicted 
that  they  plan  to  make  use  of  the  libraries  as  soon  as  they  are  available.  On  the 
basis  of  the  web  site  and  demonstrations,  we  will  be  giving  an  invited  presentation 
on  variable  resolution  displays  at  the  Society  for  Information  Display  meeting  in 
San  Jose  next  summer.  The  downloadable  demonstrations  now  include  real-time 
foveation  of  static  8-bit  color  images  and  24-bit  color  movies,  where  the  user  can 
control  the  foveation  point  with  a  mouse.  In  addition,  there  is  a  real-time 
simulation  of  fixed  bandwidth  telecommunications  showing  how  foveation  can 
dramatically  increase  frame  rate  without  sacrificing  field  of  view. 

(5)  We  have  interfaced  the  current  foveated  imaging  software  with  a  512x512  8-bit 
b/w  camera  and  two  separate  eye  trackers:  an  ASL  desktop  heads-free  eye  tracker 
and  the  OWL/ASL  V8  helmet  mounted  eye  tracker.  The  software  library  contains 
appropriate  routines  for  communicating  with  the  ASL  eye  trackers.  For  most 
observers  who  have  tried  these  eye-tracking  systems  the  foveated  imaging  works 
quite  well.  However,  we  have  found  that  for  a  small  percentage  of  observers  the 
eye  trackers  are  not  able  to  track  over  a  wide  range  of  gaze  angles.  We  are 
planning  a  more  systematic  study  of  eye  tracking  with  foveated  imaging,  however, 
we  also  believe  that  there  are  a  number  of  applications  where  simpler  pointing 
devices  will  be  adequate. 

(6)  Two  papers  have  been  published  on  foveated  imaging,  one  describing  the 
Phase  I  work  and  one  describing  some  of  the  Phase  II  work.  A  short  paper  will  be 
published  in  the  SID  proceedings  describing  some  of  the  most  recent  work. 

(7)  The  University  of  Texas  has  filed  for  a  patent  on  the  foveated  imaging 
algorithms. 

(8)  We  have  made  progress  in  demonstrating  the  value  of  our  variable  resolution 
software  in  several  applications  including  telecommunications,  image  retrieval 
and  3D  simulation.  For  example,  we  are  currently  working  on  a  video 
telecommunications  test  bed  (with  University  equipment  funds).  Also,  we  have 
begun  creating  a  demonstration  of  how  foveated  imaging  can  be  used  to  speed  3D 
graphics  simulation  and  rendering  in  the  OpenGL  environment. 
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